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Abstract. Peer feedback can engage the learning process, but collecting the survey 
data into a usable format can be time-consuming, which can deter classroom teachers 
from undertaking the kind of in-depth analysis for classroom research. To overcome 
this detriment, and in orderto research the effects of learning by assessing, we created 
a peer feedback module (“add-on”) to a free, open-source student response system 
(SRS). Using this SRS, teachers create a survey for peer feedback. Students can 
access this survey through any web browser on any kind of device, from computers 
to smartphones, to rate their classmates’ perfonnances. The survey is reused for 
each feedback session. This module can output data files compatible with MFRM 
(many-faceted Rasch measurement) software, which allows teachers to analyse the 
collected data in a very short amount of time. 
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1. Introduction 

Teachers often use peer feedback surveys to help students focus on their classmates’ 
speeches, writing samples, poster sessions, and other types of student performance. 
This process can actually engage students so that they learn while assessing. 
However, collecting, collating and inputting this data into a digital format for in- 
depth analysis is a very time-consuming endeavor, discouraging for teachers who 
are hard-pressed for time. The Peer Assessment (PA) module of MOARS (MObile 
Audience Response System) (Pellowe, 2010) was designed to overcome many of 
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the time-consuming hurdles facing teachers and researchers who are interested in 
peer assessment. 

Typically, when engaging in peer feedback, students receive a rubric that lists 
several areas to be assessed, with scales and descriptions for each level of the scale. 
Each student will review several other students’ work, choosing the most accurate 
descriptor within each area of assessment. Each student’s data is collated and later 
made available to him or her. 

One way to collect a large amount of data is through scan sheets. Students mark 
the scan sheets, which are later fed through scan machines that can read the scores 
off of each sheet, collect the data from all of the students, and make it available 
in the form of an Excel spreadsheet. Even though this method represents a huge 
improvement over the days when one had to enter all of the data by hand into a 
computer, it is still time-consuming. 

Using online survey tools also becomes problematic. For students to give individual 
feedback to each other, one would need to create a survey for each student, which 
is an unrealistic requirement with large classes, not to mention combined classes. 
Furthermore, coordinating and distributing the web URLs for each student would 
be quite a challenge. 

2. Method 

2.1. Overview of the peer assessment module design 

The PA module for MOARS addresses these problems. MOARS is an open-source 
SRS designed to be used even with the most primitive browsers of old mobile 
phones, though it is optimized for smarter devices, such as the iPod Touch. The 
framework of MOARS allows for the devolvement of specialized modules. With 
the peer feedback module, teachers create a survey for peer feedback. This survey 
will be recycled by the system so that each student can be assessed using a new 
version of the same survey, relieving teachers of the need to create multiple digital 
versions for each student. Students are able to access these surveys with even the 
most basic of web browsers; they could, for example, do their evaluations on their 
own smartphones. 

When MOARS is first installed, teachers are added to the system, course titles 
are added and assigned to teachers, and finally, students are added and assigned 
to classes. Students receive their individual usernames and passwords from their 
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teacher, and then they can use MOARS for quiz-type and survey (questionnaire) 
activities through the web browsers of their mobile devices. 

2.2. Using the peer assessment module 

Using the PA module is very straightforward. The teacher first creates a survey, 
with questions and responses matching the rubric that the teacher wishes to use for 
the PA activity. Then, the teacher opens the PA module on her MOARS system. 
She selects the survey, then selects which class(es) are to be assessed. Various 
options exist for the title of each survey (for example, should the surname be first, 
followed by the given name?). When the teacher is ready, the system will create an 
individual shortcut for each student’s peer assessment survey. So, there is no need 
for the teacher to make more than one “master” survey. Even after the shortcuts 
have been assigned, the survey questions and options can be edited, should a typo 
or mistake be discovered. 

The list of student names and corresponding shortcuts can be exported into an 
Excel sheet. If desired, this allows students to inc lude their shortcut on their work. 
For example, a piece of writing to be assessed could contain the shortcut on its 
pages. A poster could include the shortcut on the poster itself. Presentations could 
inc lude the shortcut in the corner of each PowerPoint slide. 

The teacher has two options available concerning student access to the data, and 
both are controlled by “yes/no” toggle switches. One, should students be able to 
see the results of peer assessment after they submitted their scores? Two, should 
students be able to see their own peer assessment results? Some teachers may select 
to withhold all results until after the peer assessment session has ended. When this 
option is enabled, students can see their own peer assessment feedback as a series 
of bar charts. 

From the teacher side of the Peer Assessment module, the results are available 
in two formats, “classroom results” and “research data”. In the classroom results 
section, teachers see bar charts for each student, showing the distribution of ratings 
on each question. This data can be exported as a CSV file (which can be opened in 
Excel or other spreadsheet programs). Student rankings are also available. 

In the research data section, the teacher can create and export data files that are 
compatible with MFRM software such as FACETS (Linacre, 2010). The “Facets_ 
Rating Specification” file and the “Facets_data” files should both be saved to 
the same folder. The “Facets Rating Specification” file contains the instructions 
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(specifications) telling the FACETS software how to read the data file, and the 
“Facets data” file contains the data. This allows teachers and researchers to 
perform in-depth analysis on the data that was collected. 

3. Discussion 

The feasibility of the PA module was demonstrated in a pilot study (Flolster & 
Pellowe, 2011), and it is being used in ongoing research by the authors on peer 
assessment and learning by assessing (Flolster, Lake, & Pellowe, 2014; Flolster, 
Pellowe, Lake, & Flahn, 2013). Also, its use as a classroom tool by teachers 
simply interested in the teacher data, rather than research data, has been explored 
(see for example Brown, 2013). On the negative side, the underlying MOARS 
itself is not easy for non-technical people to install. The system requires an 
Apache web server running PF1P and MySQL. Also, every rater using the system 
has to have a username and password, making this system impractical for ad hoc 
use. 

4. Conclusions 

The Peer Assessment module for MOARS can be used by both teachers and 
researchers interested in using peer assessment in the classroom. A single survey 
can be created and then reused for each member of even very large groups of 
students, each with a personalized title for the survey. Teachers have fast access to 
classroom data, and researchers can quickly produce files for analysis by MFRM 
software. MOARS and the PA module are both open source software available for 
free download (Pellowe, 2010). 

Acknowledgements. We would like to thank Robert Chartrand at Kurume 
University for providing an iPod Touch classroom for early versions of MOARS to 
be trialled, as well as the students and faculties of our respective universities where 
this system has been used. We would like to thank the early users of MOARS for 
their comments and suggestions. 

References 

Brown, I. (2013). Teaching languages with mobile devices. Paper presented at the JALT 2013 
International Conference, Kobe, Japan. 

Holster, T. A., Lake, J., & Pellowe, W. (2014). Many-faceted Rasch analysis of peer-assessment 
as a diagnostic tool. Paper presented at the 36th Language Testing Research Colloquium, 
Amsterdam, The Netherlands. 


310 


Peer assessment in the classroom using mobile devices 


Holster, T. A., & Pellowe, W. (2011). Using a mobile audience response system for classroom 
peer assessment. Paper presented at theJALT CALL 2011 Conference, Kurume, Japan. 

Holster, T. A., Pellowe, W., Lake, J., & Hahn, A. (2013). Learning by assessing in an EFL 
writing class. In Q. Zhang & H. Yang (Eds), Pacific Rim Objective Measurement Symposium 
(PROMS) 2012 Conference Proceedings (pp. 93-108). Germany: Springer. 

Linacre, J. M. (2010). Facets (Version 3.67.0) [software]. Retrieved from http://www.winsteps. 
com/facets, htm 

Pellowe, W. (2010). MOARS (Version 0.8.3) [software]. Retrieved from http://moars.com 


311 


